Motion Analysis by Synthesis: Automatically Annotating Activities in Video

An important vision problem is to automatically describe what people are doing a sequence of video. This problem is difficult for many reasons. First, tracking the articulations of people is difficult because arms and legs are small and can move quite fast. Secondly, one must sew the estimated poses from each frame together into a coherent motion path; this is difficult because people can move in unpredictable ways. Finally, one must describe the motion with some activity annotation. This appears difficult for everyday motion since there may not be a canonical vocabulary of motions. We describe a fully automatic system that labels the activities of multiple people in a video sequence. The system decouples the choice of annotation vocabulary from the analysis procedure, allowing for easy revision of the vocabulary. The system first uses a pictorial structure model to independently detect 2D poses in each frame. The system then synthesizes 3D motion clips that looks like the 2D motions by matching poses to a stored library of motion capture data. The motion capture data is labeled offline with a class structure that allows for multiple annotations to be composed; one may walk, while waveing, for example. The lack of a canonical vocabulary also makes it difficult to evaluate experimental results. We introduce a mutual information criterion that allows one to evaluate different annotation systems given labeled test footage. We demonstrate and evaluate our system on real sequences of multiple people interacting and commercial shots from a feature-length film.